Pay-as-you-go Feedback in Data Quality Systems

In many domains such as the web, sensor networks and social media, sources often provide conflicting information. It is of utmost importance to resolve conflicts and identify correct information. A number of approaches, referred to as truth finders, have been proposed recently. They address the problem of truth discovery using different principles such as link analysis, Bayesian modeling and reputation systems. None of the existing approaches, however, leverages user feedback to improve the performance of these truth finders. In the present work, we propose a novel framework based on the concept of the value of perfect information that orders existing conflicts by their ability to boost the collective performance of the truth finder on all objects. We devise a number of algorithms that take into account the voting network structure and the level of agreement/disagreement among sources, and produce effective orderings of objects for validation with interactive response rates. Finally, we present an extensive experimental evaluation where we show that our solution outperforms existing truth finders, and also study the trade-offs between the efficiency and effectiveness of the various ordering algorithms.